Bayes Factors and BIC Comment on Weakliem

Weakliem agrees that Bayes factors are useful for model selection and hypothesis testing He reminds us that the simple and convenient BIC approximation corresponds most closely to one particular prior on the parameter space the unit information prior and points out that researchers may have di erent prior information or opinions Clearly a prior that represents the available information should be used although the unit information prior often seems reasonable in the absence of strong prior information It seems that among the Bayes factors likely to be used in practice BIC is conservative in the sense of tending to provide less evidence for additional parameters or e ects Thus if a Bayes factor based on additional prior information favors an e ect but BIC does not the prior information is playing a crucial role and this should be made clear when the research is reported BIC may well have a role as a baseline reference analysis to be provided in routine reporting of research results perhaps along with Bayes factors based on other priors In Weakliem s table examples BIC and Bayes factors based on Weakliem s preferred priors lead to similar substantive conclusions but both di er from those based on P values When there is additional prior information the technology now exists to express it as a prior probability distribution and to compute the corresponding Bayes factors This can be done for a wide range of families of statistical models Prior assessment is facilitated by de ning a parsimonious family of prior distributions and a reference set of priors can be de ned for sensitivity analysis The integrals needed to compute Bayes factors can often be evaluated almost exactly using the Laplace method The GLIB software automates much of this process for generalized linear models which include linear regression logistic regression and log linear models Weakliem considers a much analyzed cross national social mobility data set and discovers two new models for it He contends that the fact that previous researchers who used BIC failed to discover these models re ects badly on BIC However BIC strongly favors the model that he prefers so this seems to be a non sequitur especially as other researchers who did not use BIC did not discover these models either With complex observational data it is important not to stop the model selection process just because BIC favors one model over another but to continue searching for better models using formal and informal diagnostic checking and residual analysis methods as long as substantial amounts of deviance remain to be explained or the current best model seems overparameterized I would argue that Bayes factors should remain the nal criterion for model comparison